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ABSTRACT 

The use of standardized tests and test data to detect 
and address differences in cognitive styles is advocated here. To 
this end, the paper describes th& componential theory of intelligence 
addressed by Sternberg et. al. This theory defines the components of 
intelligence by function and level of generality, including: (1) 
metacomponents: higher order processes involved in decision-making, 
planning, and evaluation; (2) performance components: processes used 
to manipulate and transform information in the performance of a task; 
(3) knowledge-acquisition components: processes involved in learning 
new and contextual information; and (4) general, class, or specific 
level of generality. The paper also reviews the literature on 
identification of specific components associated with specific 
cognitive processes and the trainability of those processes. 
Following the definition and description of these information 
processing constructs, their use in both constructing test items and 
analyzing student outcomes from standardized tests is advocated. Such 
qualitative test analysis could possibly then be used, not just to 
describe or predict performance, but also to detect differences in 
cognitive styles and diagnose and remediate areas of weakness. 
( JGL) 
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Toward a Qualitative Analysis of Standardized Tests 
Usi nq an Inf ormati on Processi nq Model 

El eanor Armour-Thomas 
Yale University 

The educational products requirements of the nati on 
are more frequently being defined in terms of our 
capability to provide postsecondary educational 
opportunities for the majority of our youth, and 
a continued program of learning for most of our 
citizens. If this trend continues, selection and 
prediction can no longer be allowed to dominate 
in the technology of psychoeducat ional appraisal; 
rather, the stage must be shared, with an emphasis 
on description and prescr i pti on (i.e., the 
qual i tati ve descri pti on of i ntel lecti ve f uncti on 
leading not to the selection of those most 
likely to succeed, but to the prescription of the 
learni ng experi ences requi red to more adsquatel y 
ensure that academi c success is possibl e (Edmund 
W- Gordon, 1976, p. 33) 

This quotation, written almost a decade ago is even more 
relevant today at a time when there is a growing public concern 
that the quality of education in our nation's schools seems not 
to reflect the kinds of knowledge and competencies th^t students 
will need for life in the 21st century. Part of our dis- 
satisfaction with traditional standardized tests then as it is 
now, stems from the impression that they serve very limited 
pedagogical purposes. Test data provide us with general in- 
formation with respect to positive, neutral or negative impact 
of educational treatments, but offer minimal information on 
specific aspects of achievement or failure to achieve and thus 
do not encourage efforts at relating specific aspects of 
achievement to specific aspects of treatment. Test data arc often 
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used to inform the decision making process with respect to the 
selection of students for entrance into educational programs and 
to predict their performance when exposed to these opportunities. 
However, they are limited in their utility for the conduct of 
instruction or guidance of learning. At best, test data analysis 
provide gross characterization of success and failure of students 
in relation to some reference group (e.g., percentile rank or 
grade equivalent) but they tell us little about the adequacy of 
performance, are insensitive to differential response tendencies 
and do not reflect the process by which achievement is developed 
(Gordon, 1976). 

The position being advanced here is that the generation 
and interpretation of test data should serve diagnostic 
functions more adequately so that pedagogical intervention may 
be more responsive to functional differences in children's 
performance. Test analysis should reveal the implicit 
cognitive processes embedded in items and to describe student 
achievement in terms of efficiency and accuracy of process 
executions furthermore, test data analysis should determine 
the extent to which students use efficient strategies or 
whether they represent information optimally in any given 
task of interest. Recent advances in psychological theory of 
learni ng and cogni ti on poi nt to the f easi bi 1 i ty of such a 
qualitative approach to test data analysis and interpretation. 

Current research approaches use models, concepts and 
techni ques from cogni ti ve psychol ogy to anal yze compl ex 
performance i n i ntel 1 ecti ve tasks and to i denti f y the cor sti tuent 



cognitive processes and strategies in efficient and inefficient 
learners. This paper reviews the theory and research on the 
identification and trainibility of cognitive processes with a 
view towards a reconceptual ization of standardized test data 
analysis that reflects such developments. 
Theoret i cal Per spec t i ve 

Cognitive theorists, especially those who espouse an 
information processing perspective of intel 1 igence, propose that 
in part, the psychological bases of intelligent behavior can be 
understood in terms of components of information processing 
( Carroll, 1976; Campione and Brown, 1979; Hunt, 1978; Jensen, 
1979; Newell and Simon, 1972; Pellegrino and Glaser, 1979; 
Snow, 1979; Sternberg, 1977, 1979). Although they consider 
different components to account for the latent abilities that 
underlie intelligent behavior, they all view the "component " 
as the basic unit for understanding individual differences in 
intelligence. (Sternberg, 1977, 1979), described a "component" 
as an elementary information process that operates upon an 
internal representation of objects or symbols. Sternberg's 
theory of intelligence will ser </e as the conceptual framework 
out of which will evolve the information processing model 
for the analysis of test data. 

In his componential theory of intelligence, (Sternberg, 
1977, 1979, 1980), subdivides components of intelligence in two 
ways: by function and by level of generality. 
Classification of Components by Function 

Five different types of components can be distinguished on 



the basis of function; Met acomponents - higher-order processes 
involved in decision making and planning daring problem solving 
and evaluating the success of one's problem solving performance. 
He con si ders six met acomponents as of particular importance i n 

any problem solving situation: 

(a) def i ni ng the nature of the problem 

(b ) sel ecti ng the components or steps 
needed to solve the problem 

(c) selecting a strategy for ordering 
components of probl em sol vi ng 

(d) selecting a mental representation for 
information in the problem 

(e) allocating resources (sett i ng speed- 
accuracy tradeoffs i n probl em sol vi ng 

(f > monitoring solution. 

Performance components- processes used to manipulate and 
transform information in the actual performance of a task. While 
processes may differ depending on the nature of the problem, in 
general, they are organized in four phases of problem execution: 

(a) encoding stimuli (storing information in 
worki ng memory and retr i evi ng i nf ormati on 
from long term memory that might be 
potentially relevant for interpreting the 
stimul i 

(b) combining elements into a workable 
strategy 

(c) just if yi ng one option as preferred al though 
not ideal 

(f) making a response 

Knowl edQe-acqui si ti on components . These processes are i n vol ved i n 
learning npw and contextual information. Knowledge-acquisition 
related processes store, maintain and retrieve information 
upon which performance components operate. Examples of these 
components can be found in reading comprehension passages which 
require an individual to sift relevant from irrelevant in- 
formation, combine disparate pieces of information into a 
meani ng-f ul integrated whol e or to compare old and new 



information in figuring the meaning of a word. 
Classification of components by level of generality 

Three different types of components: general, class and 
specific can be distinguished on the basis of generality. 
General components are those processes involved in the 
execution of all tasks common to a universe of tasks under 
consideration. A metacomponent such as def i ni nq the nature of 
the problem and a performance component as encodi nq 
constitute examples of general components on an aptitude test. 
Class components are those processes involved in a subset of 
tasks nested within a broader task universe. Consider for 
example, the subset of verbal inductive reasoning tasks found 
on any Scholastic Aptitude Test. Inference- a process used in 
finding the relationship between the first two terms of an 
analogy and application - the process used in carrying over 
the inferred relation to the last two terms of the analogy to 
establish an ideal solution, would constitute class components. 
Speci f i c components are those unique processes involved 
in the execution of single tasks within any given task universe. 

The di f f erent types of components and thei r level of 
generality are theorized to be closely interrelated. For those 
components distinguished on the basis of function, metacomponents 
assume a pivotal role in an integrated intelligent system. Only 
metacomponents can directly activate and receive feedback from 
other types of components; while other components can activate 
and also receive feedback from each other, they do so only 
through the mediational filters of the metacomponents. For 




components distinguished on the level of generality, the 
pattern of interrelations will depend on the breadth of the 
universe of tasks under consideration* A process considered a 
general component in the universe of verbal inductive reasoning 
tasks might be a class component in the universe of both 
verbal inductive reasong and spatial visualization tasks. 

Cognitive process researc h 

Within the last decade, numerous studies have sought 
to identify the existence of these processes through the analysis 
of successful task performance on a variety of complex tasks. 
With respect to performance components, Sternberg (1977a, 1977b; 
Sternberg and Gardner, 1982,1983; Sternberg and Nigra, 1980; 
Sternberg and Rifkin, 1979) identified seven processes which 
underlie successful solution of verbal analogies. In many 
investigations, these processes - encoding, inferring, mapping, 
applying, justifying and responding - have accounted for 
variability in subjects* response latencies and error rates. 
Similar findings on geometric , number and verbal analogical 
reasoning tasks were reported by Pellegrino and Glaser (1982). 
Different types of performance components have also been 
identified in reading performance ( Hunt, Lunneborg & Lewis, 
1975; Jackson & McClelland, 1979; Perfetti 2< Lesgold, 1977; and 
Scarborough, 1977), problem solving in geometry (GreeRDj 1978) 
and Physics (Larkin, 1981). 



vestigated in a variety of tasks. In one of their analogies 
studies, (Sternberg & Gardner, 1933) isolated metacomponents re- 
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lated to planning. In another analogies study, (Sternberg & 
Rifkin, 1979) found that better reasoners tend to spend a longer 
time in encoding terms of an analogy than do poor reasoners- In 
yet another study, (Sternberg and Rifkin, 1982) it was found that 
older children tended to perform the processes of analogical 
reasoning more exhaustively than do younger children. Other meta- 
componential processes have been addresses in studies of verbal 
comprehension as well. Examples of metacomponential processes so 
studied include: deciding on the nature of the problem ir. reading 
comprehension (e.g., Just & Carpenter, 1980; Anderson S'. Biddls, 
1975), deciding on processes for solving a problem in reading 
( Frederiksen, 1980; Kintsch 2< van Dijk, 1978), deciding on how 
to make a mental representation of information ( e.g., Anderson, 
1976; Keil, 1979; McNamara 8< Sternberg, 1983; Sternberg & Mc- 
Namara, 1985) and solution monitoring ( Collins ?< Smith, 1982). 

The findings from these studies have led to further in- 
vestigations on the trainibility of these processes in both lab- 
oratory and applied settings and in a variety of domains. 
Although the durability and general i zabi 1 i ty of training remain 
problematic in some areas, impressive training outcomes have been 
rep or ted i n reason i ng and probl em sol vi ng (Feuerstei n, 1979) 
Holzman, Glaser & Pellegrino, 1976; Linn, 1973; Sternberg, Ketron 
£>. Powell, 1982), and memory ( Belmont & Butter-field, 1971; Brown 
& Campione, 1977; Wanschura & Borkowski, 1975). 

To sLwnmar i se, as the research rsvi ewd showed, di f f erent 
types of information processing components do underlie perform- 
ance on complex tasks and in many instances do provide a good 
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account of the variability in subjects' response latencies and 
error rates. Analysis of behavior in terms of cognitive 
constructs can yield a wealth of diagnostic and pedagogic in- 
formation to an examiner. A review of latency scores and error 
data can pinpoint the source/s of an individual's strengths and 
weaknesses in a way that would lead to a prescription for 
remedying an observed deficiency or capitalizing on an observed 
strength . Furthermore, componenti al anal ysi s can reveal the 
extent to which processes are executed efficiently or whether 
they are inaccessible or unavailable to the individual. Finally, 
process analysis can specify whether such an individual uses an 
inefficient strategy in problem solving or whether an individual 
mentally represented the problem in a suboptimal manner. Thus, a 
model stemming from such an analysis of human behavior might be 
conceptualized in terms of the following stimulus and person 
characteristics (a) type of components; (b) level of 
components; (c> type of information content (e.g., numerical, 
geometric or verbal); (d) type of strategy; (e) form of mental 
representation on which strategy end component operate. 
Componential analysis of standardized tests 



lying intellectual tasks, how might such information be used 
to improve the state of the art of standardized tests?. In our 
own work, we have begun to unbundle the process dimensions that 
seem embedded in items on standardized tests in science and 
mathemati cs; more recent 1 y, we have used a componenti al 
analytic technique to design teacher made tests and to analyze 
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examinee's behavior in an intellectual enhancement program for 

high school students. In a related fields Sternberg and his 

associates are currently working on a test of intellect based on 

his conceptions of intelligence. The claim made here is that 

information processing constructs should guide both item 

construction and analysis of examinee behavior- On a hypothetical 

standardized test for example, an information processing 

perspective might seek answers to the following questions related 

to item construction: 

( 1 > What are the task domai ns or content areas 
for which items were written? 

(2) Under which factorial categories might items 
fall and in which item response might cluster 
empirically? 

(3) Into what forms of representation are items 
written and how might they be classified 

f unctionall y? 

(4) What are the task level s for each i tern on the 
test? 

(5) What level/s of process might be ■ required for 
i terns wi thin each task 1 evsl ? 

(6) What type of process/es might each item 
require for successful task performance? 

In assessing student outcomes in terms of information processing 

constructs, the following questions might also be addressed on 

the hypothetical standardised test: 

(1) What component processes were used on each 
test item? 

(2) On what internal representati on/s did such 
processes operate? 

(3) Into what strategy or strategies did 
di f f erent components combi ne? 

(4) How consistent was the use of strategies 
by individual students? 

(5) What was the quality of response time per 
i tern? 

(6) What was the quality of response time per 
component? 

(7) How might student errors be char act eri zed? 
Satisfactory answers to these questions on item construction 




and student performance would go a long way in enabling us 
to better understand the nature and demand of tests and the 
nature and behavior of examinees. Such a focus would guide the 
design of tests that are sensitive to individual differences in 
a way that traditional tests are not. 



is that they reflect an overly selective view of achievement 
and are dysfunctional and counterproductive to the purposes 
of pedagogy (Glaser, 1977, Gordon, 1977). As these critics have 
argued, this is true of their use with all children; but when we 
use them to assess achievement of the poor, disadvantaged and the 
discrimi nateJ against, the problem is compounded. Fortunately, 
we have come to recognize that individuals differ in the efficacy 
with which they process, strategize and mentally represent 
information and test design and analysis of tests should be more 
responsive to these functional characteristics of human behavior. 
Until now, inadequate attention was given to these concerns. 
Hopefully, the theoretical insights and empirical data from 
cognitive science may be used to guide the qualitative analysis 
of tests and test-takers* behavior and may lead to the develop- 
ment of tests that would enable us to diagnose and eventually 
remedy deficiencies in observed behavior. At the present time the 
cost to develop these tests or the time to administer them is 
prohibitive and therefore not practically feasible. But 
the technical feasibility has already been demonstrated in 
laboratory investigations and it would indeed be a red letter day 
for education and the testing industry when such devel omements 
are considered to be economical ly f e\=>ible. 
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